Non-Autoregressive Coarse-to-Fine Video Captioning

It is encouraged to see that progress has been made bridge videos and natural language. However, mainstream video captioning methods suffer from slow inference speed due the sequential manner of autoregressive decoding, prefer generating generic descriptions insufficient training visual words (e.g., nouns verbs) inadequate decoding paradigm. In this paper, we propose a non-autoregressive based model with coarse-to-fine procedure alleviate these defects. implementations, employ bi-directional self-attention network as our language for achieving speedup, on which decompose into two stages, where different focuses. Specifically, given determine semantic correctness captions, design mechanism not only promote scene-related but also capture relevant details construct coarse-grained sentence ``template''. Thereafter, devise dedicated algorithms fill in ``template'' suitable modify inappropriate phrasing via iterative refinement obtain fine-grained description. Extensive experiments benchmarks, i.e., MSVD MSR-VTT, demonstrate approach achieves state-of-the-art performance, generates diverse descriptions, obtains high efficiency.